In a typical digital image capture device (for example, a digital camera or cellular telephone having digital camera functionality), an image sensor captures an image as a matrix of pixel values. The image sensor (for example, a CCD image sensor or a CMOS image sensor) may output the values as digital values or may communicate information to another integrated circuit (sometimes called an Analog Front End/Timing Generator or “AFE/TG”) that in turn outputs digital values. The digital values are often in Bayer format. The Bayer format values are often converted into a set of three tristimulus values (such as a set of three RGB pixel values). For each pixel, there is one such R (red) pixel value, one such G (green) pixel value, and one such B (blue) pixel value.
The image sensor typically uses a different color filter for each color detected. For example, a first color filter is disposed over a sensor for red, a second color filter is disposed over a sensor for green, and a third color filter is disposed over a sensor for blue. The operation of the image sensor and color filters is such if light of a pure red color is incident on the image sensor, then the image sensor outputs RGB pixel values that involve not only a red pixel value, but also involve a non-zero green pixel value and/or non-zero blue pixel value. The RGB pixel values output by the image sensor are therefore not pixel values for pure red even though pure red light was incident on the image sensor. The presence of other color components is said to be a “color error.” Similarly, if light of a pure green color is incident upon the image sensor, then the image sensor may output RGB pixel values that involve not only a green pixel value, but also involve a non-zero red pixel value and/or a non-zero blue pixel value.
So called “color correction” is therefore performed on each set of RGB values as output from the image sensor in order to convert the RGB sensor output values into “corrected” RGB values. FIG. 1 (Prior Art) is a diagram that illustrates a set of three unconverted RGB pixels values in a column vector at the right. The “R” represents a red pixel value. The “G” represents a green pixel value. The “B” represents a blue pixel value. The column vector is multiplied by first 3×3 conversion matrix using matrix mathematics to generate a corresponding set of three converted R′G′B′ pixel values in a column vector at the left. The “R′”, “G′” and “B′” represent the red, green and blue pixel values as converted (i.e., corrected).
FIG. 2 (Prior Art) illustrates an example where pure red light is detected as a set of three RGB pixel values of (100,50,50). The “100” in this notation indicates an intensity of red. The next “50” indicates an intensity of green. The last “50” indicates an intensity of blue. Note that there is significant color error in that there are significant components of green and blue in addition to red. The first 3×3 matrix is applied such that a set of three R′G′B′ pixel values of (200, 0, 0) is output. Note that the correction works properly in that there is no green or blue component in the resulting converted pixel values (200, 0, 0).
FIG. 3 (Prior Art) illustrates how three different sets of RGB pixels values are converted using the first matrix. The “#1” above the arrow represents use of the first matrix. The uppermost conversion is the conversion illustrated above in FIG. 2 for a condition where pure red light is incident upon the image sensor. The next lower conversion is a conversion for a condition where pure green light in incident upon the image sensor. The resulting pixel values (0, 200, 0) is corrected in that they involve no red component or blue component. The bottom-most conversion is a conversion for a condition where pure blue light is incident upon the sensor. The resulting pixel values (0, 0, 200) is corrected in the that the values involve no red or green component.
FIG. 4 (Prior Art) illustrates two transformations. The upper transformation illustrates the transformation of pixel values (75, 50, 75) output by the image sensor when pure magenta light is incident upon the image sensor. The red pixel value and the blue pixel value are both 75. The value is therefore said to be “balanced”. When the first matrix is applied, the result (100, 0, 100) properly involves equal intensities of the red and blue components, and no component of green. A typical image sensor may, however, not necessarily output balanced pixel values if pure magenta light is incident upon the image sensor. The image sensor may, for example, output an unbalanced value of (75, 50, 70). The red and blue components in the unbalanced value are not identical.
The lower transformation illustrated in FIG. 4 illustrates the result (105, 5, 85) when the first matrix is applied in an attempt to color-correct the unbalanced (75, 50, 70) pixel values. The (105, 5, 85) result is incorrect in that the intensities of red and blue differ, and also in that there is a non-zero amount of green.
FIG. 5 illustrates a second matrix. The unbalanced (75, 50, 70) pixel values are multiplied by the second matrix to output a “corrected” value (100, 0, 100). The result is correct in that the red and blue components are equal, and there is no green component.
To perform a color correction operation on an image having many sets of RGB pixel values, each set of three RGB pixel values is treated separately. If the RGB pixel value is one of the sensor output values that would have resulted were pure red, pure green, or pure blue to have been incident on the image sensor as indicated in FIG. 3, then the first matrix is applied to perform color correction on the RGB pixel values. If, on the other hand, the set of RGB pixel values is the set of unbalanced sensor output values that would have resulted if magenta were incident upon the image sensor as indicated in the bottom example of FIG. 4, then the second matrix is applied. A decision is therefore made, on a pixel-by-pixel basis, as to which matrix is to be used to perform the color correction.
FIG. 6 illustrates three additional transformations. The upper transformation illustrates a transformation of balanced RGB pixel values (75, 75, 50) that would be output from the image sensor if pure yellow light were incident upon the image sensor. The RGB pixel values are “balanced” because the intensities of red and green are identical. If the first matrix is applied, then a proper RGB value of (100, 100, 0) is output as indicated by the uppermost transformation. Again, an image sensor may not output a balanced value when pure yellow is incident upon it. The image sensor may, for example, output an RGB value of (75, 70, 50). As the second transformation of FIG. 6 illustrates, applying the first matrix to such an unbalanced value results in incorrect RGB values (105, 85, 5). The red and blue components are not equal, and there is a small amount of blue. As the third transformation of FIG. 6 illustrates, applying the second matrix also does not result in correct RGB values. The output is (80, 135, 20). The red and green components are not equal, and there is an amount of blue.
FIG. 7 (Prior Art) illustrates a third matrix that properly converts the unbalanced yellow sensor pixel values (75, 70, 50) of FIG. 6 into a converted RGB pixel values (100, 100, 0). The red and green components are equal, and there is no blue component. Accordingly, for each set of RGB pixel values, a determination is made as to which one of the three matrices (matrix 1, matrix 2, or matrix 3) will be used. Using the three matrices of these examples, however, it is not possible to perform several transformations that might be necessary to perform accurate color correction. FIG. 8 (Prior Art) illustrates three such transformations that cannot be performed using the three exemplary matrices. A significant number of matrices therefore may be employed to perform adequately accurate color correction in a digital image capture device.
FIG. 9 (Prior Art) is a diagram that illustrates a use of six different matrices to perform so called “color correction” in one prior art digital camera. The two dimensional diagram illustrates the YCbCr color space. Each pixel involving a set of three RGB pixel values can be converted using a well known conversion matrix into another set of three YCbCr pixel values. The YCbCr pixel values are said to be in the YCbCr “color space” whereas the RGB pixel values are said to be in the “RGB” color space. Color and luminance information about a pixel can be represented by a set of three RGB pixel values and can also be represented by a set of three YCbCr pixel values. In the YCbCr format, the Y represents brightness (or luminance) of the pixel. The Cb and Cr values define the color (or chrominance) of the pixel. The two dimensional diagram of FIG. 9 therefore represents a graph of all possible chrominances that a pixel can have. The color space of the diagram is sectioned into six areas 1-6. The boundary between areas 1 and 2, for example, is defined by the blue and red pixel values of a pixel being identical (B−R=0). The boundary between areas 1 and 6, for example, is defined by the green and blue pixel values of a pixel being identical (G−B=0). Pixels falling within area 1 have values where R−G>0, G−B<0 and B−R<0. In this way, for a given set of RGB pixel values, a determination is made as to which area the RGB pixel value belongs. A different matrix is applied for each area. The color space sectioning of FIG. 9 is performed as a way to determine which one of six matrices should be applied to achieve the best color correction.
Consider a situation in which a first set of three uncorrected RGB pixel values is disposed at location 7 close to the boundary where B−R=0. B−R for the set of RGB pixels is positive, but only slightly so. The set of uncorrected RGB pixels is determined to be in area 1, so a first matrix is applied to perform color correction. Next, consider a situation in which a second set of three uncorrected RGB pixel values is disposed at location 8 close to the boundary where B−R=0. B−R for the set of RGB pixels is negative, but only slightly so. The set of uncorrected RGB pixels is determined to be in area 2, so a second matrix is applied to perform color correction. Even though there is only a slight difference between the values in the first and second sets of uncorrected RGB pixel values, entirely different matrices are applied to the two sets of pixel values. The result of the color correction operation using the scheme of FIG. 9 is an undesirable jump or disparity in the color corrected RGB output values when the uncorrected RGB input values only exhibit a slight change, one with respect to another. A solution is desired.